Automatic Facial Mask Rule Violation Detection using Keras & Tensorflow to Limit COVID-19

Abstract

COVID-19 is a pandemic disease that spread by itself coming in the contact of people. It was initially started from China and now it has been spread all over the world and many casualties have been occurred. Social distancing commonly known as physical distancing is a non-pharmaceutical approach through which it can be reduced. But social distancing only works when people started wearing mask because it can spread by sneezing even having distance among people. So wearing mask is mandatory to stop spreading this virus at its possible extent. In this paper, it has been intended to identify the people who are wearing mask or not. By the help of CCTV camera it can be recognized at the entrance of various public places such as mall, airport, railway station, mart and many more. If facial mask can be recognized effectively with high level of accuracy then it can become mandatory for people who are violating the rules. The proposed system uses Keras and Tensorflow model for identifying whether people are following the rule or not. Tensorflow is a deep learning methodology through which facial mask can be detected with all kind of situations. Proposed system is able to classify whether a person wear a mask or not, it is also able to identify whether people incorrectly wearing mask i.e. partial wearing. It is mandatory to identify whether people are properly using the mask or not. System identify this kind of situation and classified them accordingly. System uses hybrid technique by combining two algorithms i.e. keras and tensorflow. By combining both the systems it can be identified more precisely to identify the rule violations.

Introduction

I. INTRODUCTION

As per the World Health Organization (WHO) official Circumstance Report – 205, coronavirus 2019 (COVID-19) has universally tainted more than 20 million individuals causing over 0.7million death [1]. People with COVID-19 had a wide extent of side effects and its indications to genuine sickness. Respiratory issues like windedness or trouble in breathing is one of them. Older individuals having lung infection can have genuine difficulties from COVID-19 sickness as they have all the earmarks of being at higher danger [2]. Some human pertain coronaviruses that taint from 229E, HKU1, OC43, and NL63. Prior to crippling people, viruses like 2019-nCoV, SARS-CoV, and MERS-CoV creatures [3]. People having respiratory issues can wear mask and stay away to protect themselves. Environmental factors of an infected individual can cause contact transmission as drops conferring. To check certain respiratory viral illnesses, including COVID-19, wearing a clinical cover is exceptionally important. The general population ought to know about whether to put on the veil for source control or abhorrence of COVID-19. WHO focuses on clinical mask and respirators for health care assistants [4].

Hence, face cover identification has turned into a significant assignment in present worldwide society. Face mask detection includes in identifying the area of the face and afterward deciding if it has a mask on it or not. The issue is generally related to general item location to identify the classes of articles. Face recognition completely manages the cluster of particular pattern of elements for example Face. It has various applications, like auto driving, education, observation, etc. Here the primary test of the assignment is to recognize the face from the picture accurately and afterward distinguish that it has a cover on it or not. To perform the observation, the proposed technique ought to recognize a face with the movement of mask. The fast advancement of computer vision has drawn more for worldwide scourge Covid-19 to empower human-computer network and further develop general health administrations. Because of the quick spread of the Covid-19, different nations are confronting a significant health emergency. As perscripted by the World Health Organization (WHO) a powerful method to shield individuals from Covid-19 is to wear clinical masks in open regions. It is truly challenging to physically screen individuals openly and distinguish the face mask in the video which is principally on the grounds that the actual mask goes about as an impediment to the face discovery calculation, in light of the fact that there are no face signs in the mask region. Accordingly, programmed face mask identification framework assists specialists with distinguishing individuals who might be powerless to contaminations illness [5].

II. RELATED WORKS

There are numerous researches were completed within side the discipline of automated face mask detection. This paper is intended to review those researches and find out the common challenges with them. The accuracy of the system is often important because correct recognition can better emulate the system with correct classification and recognition. Wei Bu et al. [6] proposed deep learning based calculation for masked face recognition. This calculation depends on a recently planned CNN network system comprises of three CNNs. They proposed a new dataset called "MASKED FACE dataset" which have 160 pictures for training and 40 pictures for testing. To conquer the overfitting problem due to the inadequate of preparing tests, they pre-train the models with the extensive FACE dataset, and tweak them with the MASKED FACE. They assess the masked face discovery calculation on the MASKED FACE testing set and it accomplishes extremely acceptable execution.

Wang Jian et al. [7] proposed a mask wearing recognition technique based on the PP-YOLO calculation, builds up a mask wearing location informational index MaskData, utilizes the Tweak strategy and the EMA procedure to move learning to the first PP-YOLO, and utilizations measurement to play out the model Pack, lastly get the PP-YOLO-Mask model. The trial results show that the PP-YOLO-Mask model proposed in this paper has a Guide of 86.69%, and has accomplished great location brings about the recognition of mask wearing in open scenes. Contrasted and other standard models, the model in this paper has better exactness and thinking speed. Has a decent application prospect in the counteraction and control of the new crown pestilence.  Alok Negi et al. [8] proposed a high level investigation techniques and different methodologies that could engage specialists, researches and the medicals to recognize the perilous COVID-19 and speed it up care systems by proficiently testing huge volumes of examination.

The result is being utilized to adequately oversee, compute, figure and screen current contaminated individuals and future possible cases. Along these lines, they proposed CNN and VGG16 based deep learning models to fuse and authorize computer based intelligence based careful steps to recognize the face mask on Mimicked Masked Face Dataset. This strategy is fit for perceiving masked and unmasked faces to assist with observing wellbeing breaks, work with the utilization of face masks, and keep a safe working climate. Soniya Sahana Srinivasan et al. [9] proposed an effective resolution for screening social distancing in open regions where it is truly challenging to screen physically. Four unique modules have been produced for individual location, social distancing, face recognition and face mask characterization. The framework performs sensibly well with an exactness of 91.2% and normal F1 score of 90.79% on the marked video dataset with normal expectation season of 7.12 seconds on a 1 second video (50~90 picture outlines), where 5.24 seconds is spent on individual location. It additionally furnishes information increase methods to manage the absence of dataset locally. Isunuri B Venkateswarlu et al. [10] proposed a pre-compiled MobileNet with a worldwide pooling block for face mask location. The pre-compiled MobileNet takes a picture and produces a multi-dimensional component map. The worldwide pooling block that has been used in the proposed model changes the component map into an element vector of 64 elements. At long last, the softmax layer performs parallel arrangement utilizing the 64 provisions. They assessed the proposed model on two freely accessible datasets. The proposed model has accomplished close to the higher accuracy on DS1 and DS2 separately. The worldwide pooling block that has been utilized in the proposed model maintains a strategic distance from overfitting the model. Further, the proposed model beats existing models in the quantity of boundaries just as preparing time. Wuttichai Vijitkunsawat et al. [11] proposed the masked face discovery by examination of 3 renowned calculations of AI: KNN, SVM, and MobileNet on various situations. From the trial, it may be seen that the level of the exactness pace of the MobileNet calculation is the most elevated proficiency, with gradual precision as well as the continuous circumstance. Be that as it may, the level of the exactness of all calculations step by step drops when observing real time because of elements like camera quality, enlightenment. Truong Quang Vinh et al. [12] proposed a face mask detection tool using Yolov3 that utilizes Haar Cascade classifier for classifying the mask and face from various frames. The drawback to Haar cascades is that they will in general be inclined to false positive identifications, require boundary tuning when being applied for surmising/recognition, and just, as a rule, are not as precise as the more "present day" calculations. Mingyuan Xu et al. [13] proposed technique which depends on SSD mask calculation. SSD is Single Shot Multibox Detector. It is a procedure that is utilized to distinguish objects in pictures utilizing a single deep neural network. Fundamentally it utilized for object recognition in a picture. By utilizing a base engineering of VGG-16 Design, SSD is ready to perform other item detectors like YOLO and Quicker R-CNN in both speed and exactness. The design of SSD is given in the figure beneath. Preparing a SSD model without any preparation will require a ton of information, so here I have imported pretrained loads (Caffe Face Detector Model) utilizing OpenCV. By utilizing the mask name and non_mask mark, the jumping box information from json records is extracted. The faces from a specific picture are extricated and put away in the information list alongside its name for the preparation cycle.

Sohaib Asif et al. [14] proposed an examination that expects to utilize deep learning to consequently identify face masks in recordings. The proposed system consists of two parts. The main part is intended for face location and following utilizing OpenCV and AI, and in the subsequent part, these facial edges are then prepared into our proposed deep exchange learning model MobileNetV2 to distinguish the mask region. The proposed structure was tried on various recordings and pictures utilizing the cell phone camera. The object is to accomplish high-precision ongoing identification and arrangement. Test results show that the work proposed in this paper can adequately perceive face masks with numerous objectives and give powerful faculty observation. This exploration is valuable for controlling the spread of the virus and forestalling openness to the virus.

III. PROBLEM IDENTIFICATION

Vivek Aswal et al. [15] proposed a system which is based on RetinaFace and VGGFace2 approaches. RetinaFace is an edge detection technique through which a face structure can be extracted for identifying the face texture. It is a deep learning methodology which is very closely to the MxNet. VGG is a Visual Geometric Group that intended to identify the particular pattern like feature features and many more. But all these deep networks are more densed in structure that may complicate the network and due to that the computational efficiency has been affected that increase the time and space complexity. This paper focuses on single camera masked face detection and identification via the following two approaches: (i) single-step pre-trained YOLO-face/trained YOLOv3 model on the set of known individuals; and (ii) two-step process having pre-trained one stage feature pyramid detector network RetinaFace for localizing masked faces and VGGFace2 that generates facial feature vectors for efficient mask face verification. The dataset employed consists of real-world video examples comprising of 7 individuals with various orientations, illuminations, and occlusions. Experimental results show that RetinaFace and VGGFace2 achieve state-of-the-art results of 92.7% on overall performance, 98.1% face detection, and 94.5% face verification accuracy respectively in 1:1 face mask verification on our custom dataset.

IV. PROPOSED METHODOLOGY

Proposed work is able to recognize face mask using Tensorflow and Keras at real time with high level of accuracy. System uses OpenCV library along with python IDE that deals with best precision. System proposes an approach that comprises for correct recognition rate. Proposed system focuses on the recognition of face mask with correct usage of it as per the situations. System automatically detects if face mask is not properly applied or if it has been masked by hand. Proposed system classifies the differences among the frames that represent the frequent changes in the frames in a short interval of time. Recognizing face mask at real time may people force to use of face mask especially at mass places and prevent them from COVID-19.

Here system focuses on loading the face mask detection dataset and train a model (using Keras and TensorFlow) on this dataset, and then serializing the face mask detector. Once the face mask detector is trained, it can be then move on to loading the mask detector, performing face detection, and then classifying each face as with mask or without mask. It is a challenge where a system is to detect the facial mask whether it has been applied on the face correctly or not. It may be in partial pattern then system has to predict the situation as no mask detected or incorrect uses of mask. So, the proposed system is able to detect all kind of circumstances and return the result accordingly with higher level of accuracy.

There are two phases for facial mask detection, first one is to detect the face and if face has been detected with all the face features such as eyes, nose, mouth and jaw lines then it can be considered as the person is violating the rule and no mask has been detected. Here the system has been trained with Keras and Tensorflow with possible extent of datasets with almost all kind of situation with roation, scaling and many more challenges. Once the system has been trained for both mask classification and face classification then system is ready to predict the situation. In the second phase of the system, it is able to classify the mask from video frames or images then system targeted the region of interest and determine whether the person is with mask or not.

A. Flowchart

As per the flow chart, first of all system will attain the frame for identifying the facial mask, once the frame has been imported then system will call the trained models or network for applying to the system for recognizing the facial mask and face. Once it has been applied to the system then it will attain the feature vectors and later feature vectors get stacked or arranged for classification. Once the features have been classified then system predict the situation whether a person correctly uses the mask or not. System also classifies all the features of the face and compare to the threshold. Here threshold is a situation where it can be decisive that how a person wear his mask whether it is partial or correctly utilized. All these situation can be analysed by the system and decision has been taken accordingly. So, the system is proficient to identify the correct situation and decision has been taken effectively as per the trained model. Both keras and tensorflow are more proficient to detect the face and facial features. As per the features it can be easier to identified the face mask and violation related to the facial mask detection.

B. Keras & Tensorflow based Fask Mask Detection

Input: Datasets

Output: Feature Prediction

Step 1.  Import data

D = {xn, yn} is a dataset

Step 2. Transform and normalize data by mean, standard deviation and float to integer

Step 3.   Initialize the parameters for generating bounding boxes as the predicted detection outputs

Step 4.  Set Parameters W and b as tensorflow variables

Step 5.  Load the models Keras & Tensorflow

Step 6. Built Computation Graph

Step 7.  Compare with the threshold Situation as St

Step 8. if St > Td then

            Violation Detected;

          else

             No Violation Detected;

          end else

end if

Step 9.End

V. RESULT ANALYSIS

The result has been computed on the basis of various tests. Various frames have been tested and evaluated the information as per the facial mask rule. There are four parameters True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). If a frame contains pedestrian with following the facial mask rule and system detected it positively then it encountered in the category of true positive but if system detect the same as violating the rule then it comes in the category of true negative. If a frame contains pedestrian with violating the facial mask rule and system detected it positively then it encountered in the category of false positive but if system detect the same as following the rule then it comes in the category of false negative.

Table No. I Result Analysis for Mask Detection

Table No. II Result Analysis for No Mask Detection

Table No. III Result Comparison for Mean Value

Conclusion

This paper is effectively assessed a few frameworks which have been implemented till now where there are a few imperfections over there. Most of the system uses machine learning techniques to prepare the framework with different samples. Yet, an enormous dataset can burn-through the huge measure of memory that expands the execution time where it is imperative to convey as prior as conceivable with elevated level of precision. The paper proposed a system which is based on Keras and Tensorflow models for training the face datasets and identifying the facial mask for classifying the rule violations. System pertained high level of accuracy till now with less computational time. This paper depicts the strategy for detecting facial mask rule violations using image processing tools. So the feature extraction step is sufficient relying upon the area where the face mask is identified. Thus, proper division calculation is required which can be adequately find facial features. There are various researches met the desired accuracy but they are not proficient to detect the mask at each level or situations. Proposed system is more effective that can detect mask at every point whether it is partial or complete. In future a system can be developed that may have better accuracy rate with less false alarm or recognition that acquire less execution time.

References

[1] \"Coronavirus disease 2019 (covid-19): situation report 205\", 2020. [2] \"Coronavirus Disease 2019 (COVID-19) – Symptoms\", Centers for Disease Control and Prevention, 2020, [online] Available: https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html. [3] \"Coronavirus — Human Coronavirus Types — CDC\", 2020, [online] Available: Cdc.gov. [4] Das, M. Wasif Ansari and R. Basak, \"Covid-19 Face Mask Detection Using TensorFlow, Keras and OpenCV,\" 2020 IEEE 17th India Council International Conference (INDICON), 2020, pp. 1-5, doi: 10.1109/INDICON49873.2020.9342585. [5] \"Advice on the use of masks in the context of COVID-19: interim guidance\", 2020. [6] Bu, Wei; Xiao, Jiangjian; Zhou, Chuanhong; Yang, Minmin; Peng, Chengbin (2017). [IEEE 2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM) - Ningbo, China (2017.11.19-2017.11.21)] 2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM) - A cascade framework for masked face detection. , (), 458–462. doi:10.1109/ICCIS.2017.8274819 [7] Jian, W., & Lang, L. (2021). Face mask detection based on Transfer learning and PP-YOLO. 2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE). doi:10.1109/icbaie52039.2021.9389 [8] Negi, A., Kumar, K., Chauhan, P., & Rajput, R. S. (2021). Deep Neural Architecture for Face mask Detection on Simulated Masked Face Dataset against Covid-19 Pandemic. 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS). doi:10.1109/icccis51004.2021.9397 [9] Srinivasan, S., Rujula Singh, R., Biradar, R. R., & Revathi, S. (2021). COVID-19 Monitoring System using Social Distancing and Face Mask Detection on Surveillance video datasets. 2021 International Conference on Emerging Smart Computing and Informatics (ESCI). doi:10.1109/esci50559.2021.939678 [10] Venkateswarlu, I. B., Kakarla, J., & Prakash, S. (2020). Face mask detection using MobileNet and Global Pooling Block. 2020 IEEE 4th Conference on Information & Communication Technology (CICT). doi:10.1109/cict51604.2020.931208 [11] Vijitkunsawat, W., & Chantngarm, P. (2020). Study of the Performance of Machine Learning Algorithms for Face Mask Detection. 2020 - 5th International Conference on Information Technology (InCIT). doi:10.1109/incit50588.2020.93109 [12] Vinh, T. Q., & Anh, N. T. N. (2020). Real-Time Face Mask Detector Using YOLOv3 Algorithm and Haar Cascade Classifier. 2020 International Conference on Advanced Computing and Applications (ACOMP). doi:10.1109/acomp50827.2020.00029 [13] Xu, M., Wang, H., Yang, S., & Li, R. (2020). Mask wearing detection method based on SSD-Mask algorithm. 2020 International Conference on Computer Science and Management Technology (ICCSMT). doi:10.1109/iccsmt51754.2020.0003 [14] Ejaz, Md. Sabbir; Islam, Md. Rabiul (2019). [IEEE 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI) - Dhaka, Bangladesh (2019.12.24-2019.12.25)] 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI) - Masked Face Recognition Using Convolutional Neural Network. , (), 1–6. doi:10.1109/STI47673.2019.9068044 [15] V. Aswal, O. Tupe, S. Shaikh and N. N. Charniya, \"Single Camera Masked Face Identification,\" 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 2020, pp. 57-60, doi: 10.1109/ICMLA51294.2020.00018. [16] S. Asif, Y. Wenhui, Y. Tao, S. Jinhai and K. Amjad, \"Real Time Face Mask Detection System using Transfer Learning with Machine Learning Method in the Era of Covid-19 Pandemic,\" 2021 4th International Conference on Artificial Intelligence and Big Data (ICAIBD), 2021, pp. 70-75, doi: 10.1109/ICAIBD51990.2021.9459008. [17] D. Hunt, “Pathogenesis of tissue injury in the brain in patients with systemic lupus erythematosus,” in Systemic Lupus Erythematosus. Elsevier, pp. 341–348. [18] Z. Xu et al., “Lancet respir. med,” 2020. [19] N. H. Leung, D. K. Chu, E. Y. Shiu, K.-H. Chan, J. J. McDevitt, B. J. Hau, H.-L. Yen, Y. Li, D. K. Ip, J. M. Peiris et al., “Respiratory virus shedding in exhaled breath and efficacy of face masks,” Nature medicine, vol. 26, no. 5, pp. 676–680, 2020. [20] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014. [21] O. Cakiroglu, C. Ozer, and B. Gunsel, “Design of a deep face detector by mask r-cnn,” in 2019 27th Signal Processing and Communications Applications Conference (SIU). IEEE, 2019, pp. 1–4. [22] T. Meenpal, A. Balakrishnan, and A. Verma, “Facial mask detection using semantic segmentation,” in 2019 4th International Conference on Computing, Communications and Security (ICCCS). IEEE, 2019, pp. 1– 5. [23] M. R. Bhuiyan, S. A. Khushbu, and M. S. Islam, “A deep learning based assistive system to classify covid-19 face mask for human safety with yolov3,” in 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). IEEE, 2020, pp. 1–5. [24] A. S. Joshi, S. S. Joshi, G. Kanahasabai, R. Kapil, and S. Gupta, “Deep learning framework to detect face masks from video footage,” in 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN). IEEE, 2020, pp. 435–440. [25] Y. Wang, B. Luo, J. Shen, and M. Pantic, “Face mask extraction in video sequence,” International Journal of Computer Vision, vol. 127, no. 6-7, pp. 625–641, 2019. [26] A. Chavda, J. Dsouza, S. Badgujar, and A. Damani, “Multi-stage cnn architecture for face mask detection,” arXiv preprint arXiv:2009.07627, 2020. [27] Z. Wang, G. Wang, B. Huang, Z. Xiong, Q. Hong, H. Wu, P. Yi, K. Jiang, N. Wang, Y. Pei et al., “Masked face recognition dataset and application,” arXiv preprint arXiv:2003.09093, 2020. [28] W. Hariri, “Efficient masked face recognition method during the covid19 pandemic,” 2020. [29] S. Roy, W. Menapace, S. Oei, B. Luijten, E. Fini, C. Saltori, I. Huijben, N. Chennakeshava, F. Mento, A. Sentelli et al., “Deep learning for classification and localization of covid-19 markers in point-of-care lung ultrasound,” IEEE Transactions on Medical Imaging, 2020. [30] F. Rustam, A. A. Reshi, A. Mehmood, S. Ullah, B. On, W. Aslam, and G. S. Choi, “Covid-19 future forecasting using supervised machine learning models,” IEEE Access, 2020.

Copyright

Copyright © 2022 Eayan Francis , R. K. Chidar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.